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ABSTRACT 



A study of Hawaii's JOBS WORKS! waiver demonstration project 
determined its influence on participant self-sufficiency and receipt of Aid 
to Families with Dependent Children (AFDC) . Hawaii received a waiver to lift 
the 8-week federal limitation on upfront job search, operate on less than a 
statewide basis, require 18 hours of work, and secure matched federal funds 
for implementation. About 6,000 individuals in the Job Opportunities and 
Basic Skills Program (JOBS) were randomly assigned to experimental or control 
groups. JOBS WORKS! offered immediate job search and readiness activities and 
job development and placement services to AFDC recipients waiting for 
openings in JOBS activities. Data were from administrative records of the 
Hawaii Departments of Human Services and of Labor and Industrial Relations. 
Self-sufficiency measures included employment rates, length of employment, 
amount of total earnings, and total family income. AFDC measures included 
average amount of benefits, exit and recidivism rates, use of subsidized 
child care, and rates at which persons were sanctioned for failing to comply 
with program requirements. Results indicated that rates and length of 
employment and total earnings were significantly higher for the experimental 
group. Experimental members experienced a 6.3 percent net decrease in average 
AFDC benefits and left the rolls at significantly higher rates. However, the 
experiment had no significant effect on rates of AFDC recidivism. 

(Statistical methods and detailed results are appended.) ( YLB) 
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Executive Summary 



Background 

The State of Hawaii received an 1 1 15 waiver from the U.S. Department of Health 
and Human Services in 1994 to implement the Creating Work Opportunities for JOBS 
Clients Project. This initiative, which later became known as JOBS WORKS!, permitted 
Hawaii to: 

• Lift the 8-week federal limitations on upfront job search 

• Operate the waiver on less than a statewide basis 

• Require 18 hours of work from participants, and 

• Secure additional matched federal funds to implement the waiver. 

The Hawaii Department of Human Services (HDHS) contracted with the Center 
for the Study of Human Resources at the University of Texas at Austin (CSHR) to 
conduct an impact evaluation of the JOBS WORKS! demonstration. The pilot operated 
on the Island of Oahu from January 1995 through December 1996. 

The JOBS WORKS! Treatment 

The JOBS WORKS! demonstration offered immediate job search and job 
readiness activities, as well as job development and placement services, to AFDC 
recipients typically waiting for “openings” in education and training activities of the 
regular Hawaii JOBS program. JOBS WORKS! was based on the premise that AFDC 
adults and their families would benefit from these labor market experiences, instead of 
simply waiting for the more intensive human capital development options of the Hawaii 
JOBS program. Labor force participation, followed by education and training, was 
expected to increase income through work and reduce welfare payment in the short-term, 
while improving the long-term livelihood prospects of the participants and their families. 
A specialized unit, located in Honolulu, provided employment services to JOBS 
WORKS! participants, prior to referral to the regular JOBS program. 
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The JOBS WORKS! Evaluation 



For the evaluation, which was designed as a classical experiment, approximately 
6000 individuals in the JOBS selection pool were randomly assigned into either an 
experimental or control group. Experimental group members were called into the JOBS 
WORKS! demonstration. Control group members, who received the same treatment as 
the rest of the island’s JOBS eligible, remained idle in the selection pool until called in 
for normal JOBS treatments. 

The random assignment resulted in a experimental and control groups that did not 
differ systematically on measured or unmeasured background characteristics when they 
entered the study. Because of this equivalence between the two groups at the time of 
random assignment, any subsequently observed differences between the groups can be 
safely attributed to the effects of the differing treatment they received during the 
experiment. 

Most of the data used to evaluate the impacts of the demonstration came from 
administrative records kept by the Hawaii Department of Human Services. These records 
covered client demographic attributes, periods of AFDC receipt, program activities, 
benefits received, sanctions and so forth. Due to biases that always exist in self-reported 
employment data, all employment-related impacts were measured using unemployment 
insurance (UI) earnings records kept by the Hawaii Department of Labor and Industrial 
Relations (DLIR). 

A summary of the key findings from this analysis follows. 

Findings on Program Implementation 

JOBS WORKS! services were delivered by a single unit in Honolulu that served 
the entire island of Oahu. The plan called for a staff of eighteen workers, ten of whom 
were to be job development specialists. Staffing issues were a major constraint on 
program operations. Not all of the planned positions were filled, and turnover, training, 
and reductions in force (RTF’s) were problems. As a result of staff overload, the number 
of individuals participating in the program was smaller than expected. By the end of the 
experimental period, members of the experimental group were waiting as long for job 
search as members of the control group were waiting for education and training. Also 
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occurring during this time was a general economic slump in Hawaii. AFDC caseload had 
been on a continuing upward trend, from about 13,400 in 1989 to about 21,500 when the 
experiment began in 1995. 

The program operated as planned until the last quarter of 1995. At that time, the 
treatment afforded to the experimental group began to change. Instead of being called in 
immediately after randomization to begin job search, the control group began to 
experience ever increasing lags between randomization and activity. By the end of the 
experiment, the treatments afforded the experimental group were not very different than 
the non-treatment afforded the control group. For this reason, the analysis of the 
experiment was limited to the sample of clients randomized on or before September, 
1995. This subset of the sample included slightly more than half of the individuals 
randomized. 

The Findings on Program Impacts 

Differences in impacts between the experimental and control groups were 
calculated to judge the influence of JOBS WORKS! on participants’ self-sufficiency and 
AFDC participation. Specific self- sufficiency measures included: employment rates, 
length of employment, amount of total earnings, and total family income. AFDC 
measures included: average amount of AFDC benefits per case, AFDC exit and 
recidivism rates, use of subsidized child care, and rates at which persons were sanctioned 
for failing to comply with program requirements. 

Table I summarizes the results of these calculations. Taken together, these results 
indicate that the JOBS WORKS! demonstration significantly improved self-sufficiency 
for its early participants and reduced these persons’ dependence on AFDC. Rates of 
employment, length of employment, and total earnings were all significantly higher for 
experimental group members than control group members during the first twenty-one 
months after assignment. While the differences in total family income between the two 
groups were insignificant, this finding is not particularly meaningful because nearly every 
family in both groups had some income. Moreover, the amount of family income was 
significantly higher for experimental group members among those families with earned 



income. 



Table I. Summary of Findings 



Research Question 


Magnitude of 
estimated adjusted 
net effect 


Was estimated 
effect statistically 
significant? 


1. Did the demonstration promote self-sufficiency? 






Employment Rate 


4.5 Percent 


Yes 


Length of Employment 


1.33 Months 


Yes 


Earnings 


28.5 Percent 


Yes 


Total Family Income 






Probability that family earned income 


0.04 Percent 


No 


Amount earned for those with earnings 


8.0 Percent 


Yes 


2. Did the demonstration affect AFDC participation? 






Average Per-Case Benefits 


-6.3 Percent 


Yes 


Exits 


5.2 Percent 


Yes 


Recidivism 


-1.58 Percent 


No 


Subsidized Child Care 






Probability that SCC was received 


-2.4 Percent 


Yes 


Amount of SCC for those who received it 


1 1 .6 Percent 


No 


Sanction 


6.6 Percent 


Yes 



Source: CSHR analysis of HDHS administrative data. 



Experimental members also decreased their dependence on APDC during the 
period of study. They experienced a 6.3 percent net decrease in their average APDC 
benefits and left the rolls at significantly higher rates than control group members. 
However, the experiment had no significant effect on rates of AFDC recidivism, 
primarily due to the small sample size of persons returning to AFDC during this period. 
As expected, a significantly higher percentage of JOBS WORKS! participants received 
sanctions for non-participation. This occurred because experimental group members 
were required to participate in job search activities while most control group members 
were merely waiting to be served with no participation obligations. Child care usage was 
quite small for both groups, ranging from 4-6 percent of total AFDC recipients. 
Experimental group members used significantly less child care than control group 
members. These reasons for this finding are unclear. 

In conclusion, the JOBS WORKS! demonstration achieved most of its major 
objectives for early participants in the program. Although these early impacts are very 
positive, the shortened period of this evaluation only allowed impacts to be measured for 
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twenty-one months after assignment to an experimental or control group. Thus, it is not 
possible to predict whether these findings will hold up for a longer period of time for 
these participants or whether similar positive results will occur for later groups of 
participants. Even so, this approach shows great promise for its current status as one 
component of the Hawaii JOBS program. 



Introduction 

Background 

The State of Hawaii received an 1 1 15 waiver from the U.S. Department of Health 
and Human Services in 1994 to implement the Creating Work Opportunities for JOBS 
Clients Project. This initiative, which later became known as JOBS WORKS!, permitted 
Hawaii to: 

• Lift the 8-week federal limitations on upfront job search 

• Operate the waiver on less than a statewide basis 

• Require 18 hours of work from participants, and 

• Secure additional matched federal funds to implement the waiver. 

The JOBS WORKS! waiver demonstration project was inspired by a legislative 
audit critical of the Hawaii JOBS program and increased consensus among the Governor, 
legislators and administrators to place a greater emphasis on work than was found in the 
earlier JOBS program. JOBS WORKS! was intended to emphasize immediate job search 
and job readiness activities for non-exempt and volunteer JOBS-eligibles who were 
called in for services. The waiver was designed to expand participation and made labor 
force attachment, rather than human capital development, the centerpiece of the program. 
The demonstration was intended to quickly engage participants in work while they were 
waiting for more intensive JOBS education and training activities. The pilot operated on 
the island of Oahu from January, 1995 through December 1996. The waiver features 
have since been incorporated into the state JOBS model as the First-to Work project and 
working has become a systemwide pre-requisite for education and training. 

The JOBS WORKS! program had several objectives 

• To increase employment opportunities for clients on JOBS waiting lists 

• To provide work experience that may enhance client capacity for self-sufficiency 

• To reduce federal and state costs of AFDC grants due to increased client earnings 

• To provide a viable workforce for private sector employers 

• To improve public/private partnerships in workforce development 
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The Hawaii Department of Human Services (HDHS) contracted with the Center 
for the Study of Human Resources at the University of Texas at Austin (CSHR) to 
conduct an impact evaluation of the JOBS WORKS! demonstration. Results from this 
evaluation are summarized in this report. 

Objectives of the Research 

The overall goal of the research was to study the JOBS WORKS! process and 
statistical data related to the project and the outcomes for individuals to determine 
whether the project accomplished the goals it was intended to accomplish. The 
hypothesized goals of the project, as set forth in the waiver request, are: 

• Hypothesis A: The increased employment opportunity will result in a significant 
number of JOBS participants going to work while they wait for education and 
training, rather than being inactive. 

• Hypothesis B: The work experience gained by participants in the demonstration 
will correlate positively with increased measures of success in completing 
education and training activities and should result in the acquisition of 
employment positions that are more likely to lead to self-sufficiency. 

• Hypothesis C: The result of JOBS participants working while they are waiting 
for education and training is that the overall cost of AFDC will decrease due to 
grant reductions that are adjusted based on the earnings of the participants. 

• Hypothesis D: Implementation of this demonstration will provide a viable, entry- 
level workforce for many Oahu employers. 

• Hypothesis E: The implementation of the demonstration will help to forge a 
better partnership between business and government that will facilitate welfare-to- 
work programs. 



Research Questions to be Answered 

The goals of the research were tested by subjecting the following research 
questions to statistical analysis. The research questions represent concrete and 
measurable manifestations of the underlying goals, which tend to be more abstract and 
not directly measurable. 
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1. Does the demonstration promote self-sufficiency? Outcomes to be studied include: 

• Employment rates — For all study participants, did individuals in the experimental 
group have a higher probability of being employed than those in the control 
group? 

• Length of employment — For those employed, did individuals in the experimental 
group have a longer average length of employment than those in the control 
group? 

• Earnings — For those employed, did individuals in the experimental group have a 
higher level of earnings than those in the control group? 

• Total family income — For all study participants, did the families of the 
individuals in the experimental group have larger total family incomes than those 
in the control group? 

2. Does the demonstration affect AFDC participation? Outcomes to be studied include: 

• Incidence of AFDC benefit receipt — Was the average per-case level of AFDC 
benefits paid to the experimental group lower than the level for the control group? 

• Exit rate from AFDC — Did individuals in the experimental group have a higher 
probability of exiting AFDC than individuals in the control group? 

• Recidivism rate for exiters — For individuals who exited AFDC, was the 
probability of returning to the rolls lower for former members of the experimental 
group than for former members of the control group? 

• Use of child care — For all participants, did members of the experimental group 
have a higher probability of utilization of child care than members of the control 
group? For all participants who became employed, was this also the case? 

• Sanction rates — or all participants, was the sanction rate significantly different 
between the experimental and control groups? 



Experimental Design 

The waiver was designed as a randomized experiment. Individuals were 
randomly assigned either to an experimental group which received the experimental 
treatment or a control group which received the same treatment as the island’s regular 
JOBS population. 



Periodically during the duration of the study period, a stratified sample of 
individuals was drawn from the population of JOBS-eligible AFDC clients who were not 
already active in JOBS. The individuals in the sample were assigned to the experimental 
or control groups based on a random number generated by a computer program. The 
sample was stratified so that 65 percent of the sample would be JOBS target group 
members, 30% would be JOBS-eligible, but not in a target group, and 5% of the sample 
would be JOBS-exempt volunteers. After drawing the sample, lists of the selected 
individuals were sent to the JOBS WORKS! staff. For the first year of the experiment, 
samples were drawn monthly, but in the second year, only three samples were drawn. 
The reduction in sampling during the second half of the experiment occurred primarily 
because intake went slower than expected and because the size of the samples drawn was 
increased. 

Since the waiver was designed as a randomized experiment, the impacts may be 
measured by observing the outcomes for the control and experimental groups for a 
specified amount of time after assignment, and attributing any differences between the 
groups to the effects of the experimental treatment. This approach, referred to as a 
“posttcst-only control group design,” was required as a condition of all §1115 waiver 
evaluations by the U.S. Department of Health and Human Services. 1 This design has 
been used for several major evaluations of wel far e-to- work experiments across the U.S. 

Statistical Methods Applied 

The primary statistical tool used to analyze the differences in outcomes between 
the experimental and control group was two-sample tests for differences in means or 
proportions. For the analysis of categorical data (for example, employed versus not 
employed), the proportion in the category in the control group was compared the 
proportion in the category in the experimental group, and the difference between the two 
proportions, the estimated net effect, was subjected to a statistical test to determine if it 
was large enough that it was not likely due to chance. For continuous variables, (for 
example, the amount of AFDC benefits received), the mean of the variable for the control 
group was compared to the mean of the variable for the experimental group, and the 

1 Donald T. Campbell and Julian C. Stanley, Experimental and Quasi-Experimental Designs for Research, 
Rand McNally College Publishing Company, Chicago, 1966 
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difference between the two means, the estimated net effect, was subjected to a statistical 
test to determine if it was large enough that it was not likely due to chance. 

In addition to the basic two-sample difference tests, regressions were executed in 
which the experimental effect is measured by one of the estimated coefficients of the 
fitted regression equation. These net effect estimates have the advantage that they are 
adjusted for measurable demographic differences between the experimental and control 
groups, and they may be expected to be more precise. The adjusted net effect estimates 
from the regressions are the preferred estimates of the impact of the treatment. 

Characteristics of the Research Sample 

Table 1 shows the characteristics of the sample, tabulated separately for the 
control and experimental groups. Since this waiver was performed as a randomized 
experiment, the attributes of the experimental and control group should differ only by 
chance. The randomization scheme has been tested by comparing the attributes of the 
control and experimental groups at the point they entered the experiment. No differences 
between the mean attributes of the two groups should be noticeable. Table 1 shows the 
result of simple two-sample tests on the attributes of the experimental and control group 
subjects. The t-statistic is used to test the null hypothesis that there is no difference 
between the two groups. The table shows no significant differences between the two 
groups. This desirable outcome shows that the randomization produced comparable 
groups. 
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Table 1. Selected Characteristics of the Research Samples 



Attribute 


Mean for 
Experimental 


Mean for 
Control 


Difference 


t-Statistic for 
Ho 


Age at intake 


32.820 


33.058 


-0.238 


-1.125 


English Proficient* 


0.922 


0.914 


0.008 


1.097 


Highest Grade Completed 


10.788 


10.761 


0.027 


0.336 


High School Graduate* 


0.676 


0.683 


-0.007 


-0.567 


Male* 


0.211 


0.207 


0.004 


0.386 


Married* 


0.256 


0.256 


0.000 


0.024 


Never Married* 


0.376 


0.378 


-0.002 


-0.159 


Not Primary Individual on Case* 


0.050 


0.054 


-0.003 


-0.613 


United States Citizen* 


0.919 


0.914 


0.005 


0.722 


Dollars of AFDC benefits received 
in period prior to selection 


7063.543 


7010.797 


52.746 


0.545 


Employed in period before 
selection* 


0.334 


0.329 


0.006 


0.453 


In a target group* 


0.247 


0.237 


0.010 


0.930 


Volunteer* 


0.475 


0.465 


0.010 


0.930 


Exempt waiting to volunteer* 


0.047 


0.049 


-0.003 


-0.509 


Not Exempt, not in target group* 


0.713 


0.721 


-0.007 


-0.649 


JOBS volunteer in target group* 


0.240 


0.230 


0.010 


0.946 


Filipino ethnicity* 


o.ios 


0. 106 


-0.002 


-0.223 


Hawaiian ethnicity* 


0.275 


0.277 


-0.002 


-0.195 


Mixed race or ethnicity* 


0.225 


0.226 


-0.001 


-0.129 


White race or ethnicity* 


0.126 


0.118 


0.008 


-0.129 


Other race or ethnicity* 


0.270 


0.272 


-0.002 


-0.191 



Source: CSHR analysis ofHDHS administrative data. 

Notes: (1 ) None of the differences were statistically significant at a=5% level. 

(2) • Indicates dummy variable equal to one if individual has this attribute, zero otherwise. 



Data Sources 

The data used to evaluate this experiment came entirely from administrative 
records ofHDHS and the Hawaii Department of Labor and Industrial Relations (DLER). 
The HDHS data covered all welfare-related variables, such as AFDC benefits paid, exits 
and recidivism, sanctions, job search activities, child care, family income, demographic 
attributes of case head, and target group status. The DLER data used in this evaluation 
included unemployment insurance (UI) wage information collected from employers’ 
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quarterly UI tax returns. This data, used by DLIR to administer the unemployment 
insurance program, was used to estimate impacts for all employment-related outcomes. 

Despite the best efforts of all concerned, some observations were tainted with 
missing data. For about 0.9 percent of the subjects data could not be completely retrieved 
from the administrative databases supplied by HDHS because no observation in the 
administrative database matched the client number of the individual on the JOBS 
WORKS! roll. These unreliable observations have been deleted from all statistical 
analysis. 

Appendix A contains a detailed description of all variables analyzed, and the time 
periods for which data was collected. 



The Treatment 

The JOBS WORKS! treatment involved up-front job search and job readiness 
activity for persons on the waiting list for the JOBS program, rather than the control 
group treatment of waiting for education and training to be provided through the regular 
JOBS program. Specific elements of the treatment included: 

• Job Search. JOBS WORKS! clients participated in Job Search for 30 hours per week 
until they became employed 18 hours per week, or were closed or were transferred to 
the appropriate JOBS unit. Their hours of participation were calculated by formula. 
Under the formula, an interview with a potential employer counted as 6 hours, visits 
to employers counted for 3 hours, and mail/telephone contacts counted for 2 hours. 
Clients were expected to complete job search activity reports weekly and turn them in 
to their job developer. Actual hours per individual may be more or less than 30 
hours per week. One of the provisions of the waiver exempted the members of the 
experimental group from the federal cap of a maximum of eight weeks of up -front 
job search. 

• Job Readiness/Ho 'ala. Clients were referred at staff discretion to a job-readiness 
seminar named Ho’ ala. The 4-day, 27 hour seminar was delivered on-site by Work 
Hawaii staff. Clients were allowed a maximum of 90 minutes total absence or 
received no credit for the seminar. Reportedly there was a 50% attrition rate between 
referral and attendance and an 85% completion rate thereafter. 2 The first Ho’ala for 
JOBS WORKS! occurred in March 1995. The JOBS WORKS! Ho’ala was a 
shortened version of the 3-week, 72 hour seminar given to regular JOBS clients at the 
time. 



2 Conversations between CSHR process analysis team and Work Hawaii staff. 
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• Subsidized Child Care. Subsidized child care was provided for JOBS WORKS! 
clients who were participating in job search, job readiness, and employment. 
Approximately 5-10% of the active JOBS WORKS! caseload was eligible for child 
care, however not all eligible clients actually received child care. Authorization of 
funding for a client’s child care was controlled by the client’s job developer. The 
clients themselves were responsible for finding commercial child care or making 
arrangements with family members. Payments of up to $325 per month per child 
were made directly to client. 

• Transportation Reimbursements. Bus tokens or cash reimbursement for 
transportation expenses were issued or authorized by the client’s job developers on a 
case-by-case basis. 



Activities of Experimental and Control Groups 

Figure 1 and Figure 2 show the caseload of experimental and control subjects in 
JOBS activities over the course of the experiment. The figures require some explaining, 
since they assign each individual into only one activity per month, even though most 
individuals had more than one activity in a month. For the purposes of the figures, the 
activities were aggregated and assigned an hierarchical rank. Individuals who 
participated in more than one of the aggregate activities in a month were assigned to the 
activity with the highest rank. The aggregation and ranking scheme is summarized in 
Table 2. 



Table 2. Aggregation Scheme and Hierarchical Ranks for Aggregated Activities 



Aggregated Activity 


Abbreviation in 
Figure I and Figure 2 


HANA Activity Codes 
Included 


Hierarchical 

Rank 


Education 


EDU 


2, 3, 4, 5, 6 


7 


Employment 


JOB 


1C, ID, IE, IF 


6 


Job Search 


JBS 


10, 7A 


4 


Intake 


INT 


21,22,23,9,24,25,26,27,28 


5 


Conciliation 


CON 


41,42,43,44,45,46 


3 


Sanction 


SAN 


29 


2 


Other 


OTH 


All activities not specified for 
higher ranking aggregates. 


1 



Source: CSHR analysis of HDHS administrative data. 



While the experimental JOBS caseload reached over six hundred clients for 
several months, control group members enrolled in the regular jobs program never 
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exceeded 32 clients. At the beginning of the experiment, experimental clients were 
primarily engaged in intake and job search, as expected. As the experiment proceeded, 
the intake activity became less important as fewer individuals were outreached. Many 
experimental clients found jobs, thus increasing the employment activity. A substantial 
proportion of the experimental population failed to meet requirements, and induced an 
increase in the sanction and conciliation activities. Almost nobody in the experimental 
group engaged in education activities. 3 

As expected, the control caseload and their activities differed substantially from 
that of the experimental group. Whereas the experimental caseload enrolled in JOBS 
WORKS! rose to the six-hundred per month level, average monthly JOBS participation 
for control group members never exceeded 32. In the beginning of the experiment, more 
than half of the control group members enrolled in JOBS participated in education 
activities, and their only other activities were intake and other. As the experiment 
progressed, the share of the education caseload declines, and employment, job search, 
sanction and conciliation begin to appear. These changes in activities observed can 
probably be attributed to changes in the emphasis of the JOBS program that began in the 
second half of 1995. By the end of the period, the activities of the control group appear 
very similar to the activities of the experimental group, except for much lower sanction 
and conciliation rates for control group members. 



3 The largest caseload in educational activities was eight, which occurred in the earlier months of the 
experiment. After July, 1995, the experimental group caseload in educational activities never exceeded two 
clients. 
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Number of Cases Number of Cases 




Source: CSHR tabulation of HDHS administrative data. 



Figure 2. Control Group Caseload by Activity 




Month 



Source: CSHR tabulation of HDHS administrative data. 
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Timing of Treatments 

The main purpose of the experiment was to test whether outcomes are affected by 
having future JOBS participants execute job search activities immediately rather than 
waiting for openings in education and training activities. Accordingly, if the experiment 
is being conducted as described, one should observe members of the experimental group 
receiving JOBS activities immediately after being selected into the JOBS WORKS! list, 
whereas members of the control group should have a longer delay between selection and 
activities. 

Table 3 shows the lag between random assignment and first observed JOBS 
activity for the two groups. As may be observed from the table, in the beginning of the 
experiment, up to September, 1995, most of the experimental group received treatment 
within 60 days of random assignment, whereas the delay is much longer for the control 
group, and most control group members received no activities at all. However, from 
October 1995 onward, it becomes progressively more difficult to discern a pattern of 
difference in lag between the two groups. At least three factors contributed to this 
change. First, the backlog began to build as the job search resources became fully 
utilised. Since fewer new clients could be handled, fewer were called in. Second, as the 
experiment progressed, more and more resources had to be expended on sanction and 
conciliation, reducing the amount that could be spent on more productive activities. 
Third, to conserve computational resources, it was decided to draw larger samples at less 
frequent intervals. The original design of the experiment dictated the size of samples to 
be drawn to be based on the availability of JOBS WORKS! staff. If this dictate had been 
followed, the drawings in 1996 would have consisted of much smaller monthly samples. 
The policy of drawing larger samples at less frequent intervals and the filling up of 
available slots as the program reached its capacity caused a longer period of time to pass 
between random assignment and first activity. 

Stimulated by a legislative audit critical of the Hawaii JOBS program and 
increased consensus among the governor, legislators and administrators, structural 
changes in the state’s JOBS program in particular began in mid 1995 and affected the 
experiment from then on. The emphasis of the state’s welfare system changed from a 
long-term educational and training approach to an approach that had a greater emphasis 
on work. This change was accompanied by a decrease in funding and a concomitant 



reduction in force (RIF) at both HDHS and DLIR, including personnel at the unit 
involved in the experiment. The RIF caused increased turnover, bumping, and employee 
stress that may have contributed to limited case manager capacity. 

Because of these unintentional changes in the operation of the experiment, after 
October 1995 the group that was supposed to be doing j ob search rather than waiting for 
education and training was simply waiting for job search rather than waiting for 
education and training. Thus, CSHR researchers decided to divide the experiment into 
two periods. The first began in January 1995 and ran to September 1995, when the 
experiment was operating as planned, while the second began in October 1995 and ran to 
the end of the experiment. The statistics designed to measure the net experimental effect 
were executed on the shortened sample, because the observations from the shortened 
sample are more representative of the original intent of the experiment. To include 
observations from the less representative sample period would tend to dilute the 
estimated net effect of the experiment, because the persons in the experimental group 
were less likely to receive experimental treatment. 

A further advantage of limiting the analysis to the individuals selected between 
January 1995 and September 1995 is that this approach permits a longer time period to 
observe outcomes. The disadvantage of dropping all subjects randomized after 
September 1995 is that the sample size is decreased. Of the total 6,158 individuals with 
complete observations, 3,294 of them (53.5 percent) were included in the impact analysis. 
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Table 3. Time Lag from Date of Selection to Date of First JOBS Activity 



Experimental Group 






Lag in Days from Selection to First Activity 






Date of 


Number 


0 


30 


60 


90 


120 


150 


180 


210 


240 


270 


300 


No 


Percent 


Percent 


Selection 


Selected 


to 


to 


to 


to 


to 


to 


to 


to 


to 


to 


or 


Activity 


Active 


never 






29 


59 


89 


119 


149 


179 


209 


239 


269 


299 


More 




within 
59 days 


active 


199501 


66 


19 


17 


6 


0 


0 


0 


1 


0 


0 


0 


1 


22 


55% 


33% 


199502 


44 


24 


15 


1 


0 


0 


0 


0 


0 


0 


0 


0 


4 


89% 


9% 


199503 


121 


66 


23 


2 


1 


1 


0 


0 


0 


1 


0 


1 


26 


74% 


21% 


199504 


241 


104 


92 


4 


0 


0 


0 


1 


0 


0 


0 


2 


38 


81% 


16% 


199505 


301 


155 


101 


3 


4 


2 


0 


0 


0 


0 


0 


4 


32 


85% 


11% 


199506 


121 


54 


36 


4 


2 


0 


0 


0 


0 


0 


0 


1 


24 


74% 


20% 


199507 


131 


35 


70 


2 


1 


1 


0 


0 


0 


0 


0 


0 


22 


80% 


17% 


199508 


300 


85 


131 


8 


3 


2 


1 


0 


4 


3 


1 


17 


45 


72% 


15% 


199509 


350 


81 


155 


6 


0 


1 


0 


0 


0 


1 


2 


3 


101 


67% 


29% 


199510 


300 


43 


56 


39 


85 


9 


0 


0 


0 


0 


0 


1 


67 


33% 


22% 


199511 


351 


4 


1 


15 


53 


101 


54 


40 


2 


0 


2 


4 


75 


1% 


21% 


199512 


351 


9 


13 


7 


6 


1 


3 


55 


40 


62 


19 


5 


131 


6% 


37% 


199601 


130 


13 


8 


5 


4 


7 


7 


4 


21 


8 


0 


8 


45 


16% 


35% 


199607 


201 


13 


10 


2 


4 


1 


4 


1 


7 


5 


3 


3 


148 


11% 


74% 


199610 


121 


5 


5 


3 


6 


6 


0 


4 


6 


0 


0 


0 


86 


8% 


71% 


Control Group 


































Lag in Days from Selection to First Activity 






Date of 


Number 


0 


30 


60 


90 


120 


150 


180 


210 


240 


270 


300 


No 


Percent 


Percent 


Selection 


Selected 


to 


to 


to 


to 


to 


to 


to 


to 


to 


to 


or 


Activity 


Active 


never 






29 


59 


89 


119 


149 


179 


209 


239 


269 


299 


More 




within 
59 days 


active 


199501 


67 


2 


3 


0 


1 


0 


1 


0 


1 


1 


1 


12 


45 


7% 


67% 


199502 


41 


4 


2 


4 


1 


0 


1 


0 


0 


1 


0 


6 


22 


15% 


54% 


199503 


124 


13 


8 


6 


5 


4 


4 


5 


4 


5 


2 


11 


57 


17% 


46% 


199504 


234 


5 


2 


0 


5 


7 


3 


6 


4 


7 


6 


36 


153 


3% 


65% 


199505 


301 


1 


8 


5 


10 


2 


5 


4 


5 


3 


6 


55 


197 


3% 


65% 


199506 


111 


2 


1 


2 


2 


2 


1 


3 


3 


1 


2 


20 


72 


3% 


65% 


199507 


112 


6 


7 


2 


1 


6 


3 


9 


5 


1 


2 


6 


64 


12% 


57% 


199508 


298 


28 


23 


12 


5 


8 


6 


11 


4 


5 


5 


34 


157 


17% 


53% 


199509 


350 


0 


3 


6 


4 


2 


6 


8 


5 


5 


4 


55 


252 


1% 


72% 


199510 


300 


8 


4 


4 


1 


3 


0 


7 


5 


6 


5 


30 


227 


4% 


76% 


199511 


353 


2 


8 


4 


2 


4 


5 


3 


10 


10 


7 


45 


253 


3% 


72% 


199512 


344 


9 


10 


12 


6 


8 


8 


8 


9 


3 


7 


34 


230 


6% 


67% 


199601 


127 


19 


7 


4 


5 


5 


3 


2 


3 


0 


1 


11 


67 


20% 


53% 


199607 


199 


11 


10 


6 


2 


4 


5 


3 


5 


3 


2 


1 


147 


11% 


74% 


199610 


122 


6 


3 


3 


2 


6 


5 


4 


1 


0 


0 


0 


92 


7% 


75% 



Source: CSHR analysis of HDHS administrative data 




13 



2C 



Impacts of the JOBS WORKS! Program 

Probability of Employment 

The results of a tabulation of the employment status of the experimental and 
control groups are shown in Table 4. The tabulation is based on whether the individual 
was employed within 21 months after being selected and randomly assigned to either the 
experimental or control groups. 



Table 4. Employment Status for Experimental and Control Groups 





Employed after 
Selection 


Total Number of 
Subjects 


Experimental Group 




Number of Subjects 
Percent of Group 


901 

54.21% 


1662 

100.00% 


Control Group 




Number of Subjects 
Percent of Group 


794 

48.65% 


1632 

100.00% 










Unadjusted Net Experimental Effect 


5.56%*** 




A Hi 

4 


listed Net Experimental Effect (from 
Regression Analysis) 


4.49%*** 





Source: CSHR Estimates based on UI wage data. 

Note: *** indicates effects are statistically significant at a=0.01 level. 



Of the experimental group, 54.21 percent found employment, whereas only 48.65 
percent of the control group found employment. Thus, the experimental treatment had a 
5.56 (=54.21-48.65) percent positive net unadjusted effect on employment. The adjusted 
effect, which is adjusted by regression methods for the slight demographic differences 
between the control and experimental groups is 4.49 percent. 4 Both estimates of the net 
effect are statistically significant. 

This positive net effect was one of the expected outcomes of the experiment. It 
shows that for early participants in the JOBS WORKS! program, the work emphasis of 
the treatment resulted in higher rates of employment in the first 21 months after 
assignment to experimental or control groups. 

4 See Appendix A for an explanation of the methods used to produce adjusted net effect estimates by 
regression. 



Length of Employment 

Table 5 shows the unadjusted and adjusted net effects of the treatment on the 
length of employment for those who became employed. 5 The average length of 
employment for all subjects (experimental and control) was just over 12 months out of a 
possible twenty-one. The statistically significant net experimental effects show that the 
treatment induced an approximate ten percent positive change in the length of 
employment for those who became employed. This positive net effect on length of 
employment, together with the earlier result showing that the treatment induced a higher 
probability of employment, was one of the expected outcomes of the experiment. 

Table 5. Experimental Effects on Length of Employment 



Mean Length of Employment for 
Employed Members of 
Experimental Group 


12.73 


Mean Length of Employment for 
Employed Members of 
Control Group 


11.50 


Unadjusted Net Experimental 
Effect 


1.23*** 


Adjusted Net Experimental 
Effect (from Regression 
Analysis) 


1.33*** 



Source: CSHR Estimates based on U1 wage data. 

Note: * * * indicates effects are statistically significant at a=0.01 level. 



Earnings 

Table 6 shows the result of the statistical analysis performed to analyze the effect 
of the treatment on the amount earned by the experimental subjects who obtained 
employment. The earnings reported in the table are averaged over a 21 month period 
following random assignment. During this period JOBS WORKS! participants earned an 
average of $1,285 more than control group members, an increase of almost 15 percent. 
The adjusted net effect is even larger. After adjusting for the small preexisting 
differences between the experimental and control group, the estimate of the earnings 



5 See Appendix A for an explanation of why the sample analyzed for this part of the analysis is limited to 
individuals who became employed. 



difference of the experimental group over the control group increases to almost thirty 
percent. Both the unadjusted and adjusted net effects are statistically significant. 



Table 6. Experimental Effects on Earnings 



Mean Earnings for Employed 
Members of Experimental 
Group 


$ 9479.28 


Mean Earnings for Employed 
Members of Control Group 


$8194.18 


Unadjusted Net Experimental 
Effect 


$1285.10*** 


Unadjusted Net Experimental 
Effect as a Percent 


14.48%*** 


Adjusted Net Experimental 
Effect (from Regression 
Analysis) 


28.50%*** 



Source: CSHR Estimates based on UI wage data. 

Note: *** indicates effects are statistically significant at a=0.01 level. 



Family Income 

Unlike the earnings data which was obtained from the UI wage files, family 
income data was extracted from HDHS administrative data files. Since the family 
income data is gathered for entire families, it had to be matched to the individual-level 
JOBS WORKS! roster using case numbers. 6 Table 7 shows the experimental effect on 
the probability of having family income. Since more than 90 percent of the families had 
some income, it is not surprising that the difference between the experimental and control 
groups in this measure is insignificant. It is much more interesting to note the result 
reported in Table 8, in which estimates are displayed showing a statistically significant 
increase in family income of eight percent, for those families that had income. This 
increase in income was one of the more important outcomes expected from the treatment. 



6 During this matching process, it was discovered that in some cases, experimental group members were 
part of the same case as control group members. These “contaminated” cases were dropped from the 
family income analysis because it is not possible to unambiguously assign the family income to the control 
group or the experimental group. 
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Table 7. Experimental Effects on Probability of Having Family Income 





Had Family Income 
after Selection 


Total Number of 
Subjects 


Experimental Group 




Number of Subjects 


1516 


1637 




Percent of Group 


92.61% 




Control Group 




Number of Subjects 


1483 


1607 




Percent of Group 


92.28% 












Unadjusted Net Experimental Effect 


0.32% 




Adjusted Net Experimental Effect (from 
Regression Analysis) 


0.04% 





Source: CSHR Estimates based on HDHS administrative data. 

Note: The estimated experimental efTects were not statistically significant. 



Table 8. Experimental Effects on Amount of Family Income 





Mean Total Family 
Income for 21 
Months after 
Selection 


All Subjects with Income 


$17,326 


Experimental Group 


$18,268 


Control Group 


$16,361 


Unadjusted Net Effect 


$1,907 


Unadjusted Net Effect as a Percent 


11.01%*** 


Adjusted Net Experimental Effect (from 
Regression Aiialysis) 


8.00%*** 



Source: CSHR Estimates based on HDHS administrative data. 

Note: ••• indicates effects are statistically significant at a=0.01 level. 



AFDC Benefits Paid 

Table 9 shows the net experimental effect on amount of AFDC benefits received 
for a 21 month period after random selection. The statistically significant 6.3 percent 
decrease in AFDC benefits paid, was of the expected outcomes of the experiment, was 
brought about by two factors. First, as reported in the next section of this document, the 
experimental group achieved a higher exit rate than the control group, and those who left 
the rolls received AFDC benefits for fewer periods than those who stayed. Second, 
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because earnings offset AFDC benefits, the larger earnings of the experimental group 
tended to reduce the AFDC benefits they received. 

Table 9. Experimental Effects on Total AFDC Benefit Receipt 





Mean Total AFDC Benefits 
Received for 2 1 Months 
after Selection 


All Subjects 


$11,150 


Experimental Group 


$10,810 


Control Group 


$11,497 


Unadjusted Net Effect 


-$687 


Unadjusted Net Effect as a Percent 


■ 6 . 16 %*** 


Adjusted Net Experimental Effect (from 
Regression Analysis) 


- 6 . 30 %*** 



Source: CSHR analysis of HDHS administrative data. 

Note: Ml indicates effects are statistically significant at a=0.0 1 level. 



Exits 

Table 10 shows the net effects of the experiment on the probability of exiting 
AFDC. The probability of exii was significantly higher among the members of the 
experimental group. Exits were defined as having occurred if the individual left the 
welfare rolls for more than 45 consecutive days within 21 months of being randomly 
assigned. Exits that were followed by a return to the rolls within 45 days were not 
considered because they are not deemed an indicator of self-sufficiency. 7 

The net effect of the experiment on exits is both positive and statistically 
significant. The magnitude of the coefficient suggests that experimental group members 
were about 5.2 percent more likely to exit than their control group counterparts, other 
things equal. 



7 Short periods off the rolls usually result from failure to submit administrative paperwork in a timely rather 
than being caused by significant improvements in a client’s ability to keep and hold a job. 



Table 10. Net Experimental Effect for Probability of Exit from AFDC 





Exited after 
Selection 


Total Number of 
Subjects 


Experimental Group 




Number of Subjects 
Percent of Group 


860 

51.7% 


1662 

100.0% 


Control Group 




Number of Subjects 
Percent of Group 


759 

46.5% 


1632 

100.0% 










Unadjusted Net Experimental Effect 


5.2%*** 




Adjusted Net Experimental Effect (from 
Regression Ajialysis) 


5.2%*** 





Source: CSHR analysis of HDHS administrative data 

Note: *** indicates effects are statistically significant at a=0.01 level. 



Recidivism 

Table 1 1 shows whether experimental group members who exited from AFDC 
returned to the rolls in smaller numbers than control group members who exited. In order 
to be counted, the return to the rolls had to occur within 21 months of the individual’s 
random assignment. The regression is applied only to individuals who exited from 
AFDC. The negative 1.58 percent adjusted net effect is in the expected direction, but it is 
statistically insignificant so the treatment cannot be said to reduce recidivism for those 
who exit. 
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Table 11. Net Experimental Effect for Recidivism 





Retired 

after 

Exit 


Total Number of 
Subjects Who 
Exited 


Experimental Group 




Number of Subjects 
Percent of Group 


261 

30.35% 


860 

100.00% 


Control Group 




Number of Subjects 
Percent of Group 


241 

31.75% 


759 

100.00% 










Unadjusted Net Experimental Effect 


-1.40%*** 




Adjusted Net Experimental Effect (from 
Regression Analysis) 


-1.58%*** 





Source: CSHR analysis of HDHS administrative data. 
Note: Neither effect is statistically significant. 



Utilization of Subsidized Child Care 

Table 12 shows the result of an analysis of the probability of use of subsidized 
child care. Due to limitations on the data collection period for subsidized child care, the 
analysis is based on a fifteen month follow-up period after the date of randomization. 
The experimental group’s use of child care was expected to increase because their higher 
employment rates. However, Table 12 shows a negative and significant experimental 
effect for child care receipt of 2.4 percent. This unexpected outcome may be explained 
by referring to Table 13, which shows how many individuals used child care, and how 
much they used, broken out by whether the individual was in the experimental or control 
group, and whether the individual was employed or participated in an education activity 
during the child care follow-up period. As expected, most of the subsidized child care 
was provided to individuals who were employed. However, for reasons that are not clear, 
only 7 percent of the employed experimental group members received subsidized child 
care compared to 1 1 percent of the employed control group members. This difference in 
the proportion of employed subjects receiving subsidized child care explains how the 
experimental group could have a higher level of employment, but a lower probability of 
receiving subsidized child care. 
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Table 12. Net Experimental Effect for Use of Subsidized Child Care 





Received Subsidized 
Child Care after 
Selection 


Total Number of 
Subjects 


Experimental Group 




Number of Subjects 
Percent of Group 


73 

4.39% 


1662 


Control Group 




Number of Subjects 
Percent of Group 


107 

6.56% 


1632 










Unadjusted Net Experimental Effect 


-2.16%*** 




Adjusted Net Experimental Effect (from 
Regression Analysis) 


-2.43%*** 





Source: CSHR analysis of HDHS administrative data 

Note: *** indicates effects are statistically significant at a=0.01 level. 



Table 13. Detailed Tabulation of Use of Subsidized Child Care 



Experimental 
or Control 


Employed 
within 15 
Months of 
Selection 


In Educ. Act 
within 15 
Months of 
Selection 


Number 
of Cases 


Number 
of cases 
using 

see 


Percent 
of cases 
using 

sec 


Average 

Duration 

of 

Subsidy 

mavO 


Subsidy 
Per Case 


Total Child 
Care 
Subsidy 


Control 


No 


No 


904 


25 


3% 


184 


$1,071 


$26,782 


Control 


No 


Yes 


6 


2 


33% 


272 


$1,417 


$2,833 


Control 


Yes 


No 


717 


78 


11% 


205 


$1,504 


$117,332 


Control 


Yes 


Yes 


5 


2 


40% 


422 


$7,768 


$15,536 


Experimental 


No 


No 


828 


15 


2% 


228 


$2,261 


$33,911 


Experimental 


No 


Yes 


0 


0 


0% 


- 


- 


$0 


Experimental 


Yes 


No 


830 


57 


7% 


193 


$1,532 


$87,303 


Experimental 


Yes 


Yes 


4 


1 


25% 


302 


$3,112 


$3,112 



Source: CSHR analysis of HDHS administrative data. 



Table 14 completes the analysis for subsidized child care. Since only 5.5 percent 
of the subjects received subsidized child care, the number of observations available for 
this analysis is only 180. The average subsidy for experimental group members was 
SI 85 larger than for control group members. However, because of this small sample size, 
the estimates of the experimental effect are insignificant. 



Table 14. Net Effect on Amount of Child Care Received 





Number of 
Cases 


Average Child 
Care Subsidy 
Per Case 


All Subjects with Child Care 


180 


$1,593 


Experimental Group 


107 


$1,703 


Control Group 


73 


$1,519 


Unadjusted Net Effect 


"1 


$185 


Unadjusted Net Effect as a Percent 


■■■■ 


11.58% 


Adjusted Net Experimental Effect (from Regression 
Analysis) 


■■HU 


$115 


Adjusted Net Effect as a Percent 




7.24% 



Source: CSHR analysis of HDHS administrative data. 
Note: Effect estimates were not statistically significant 



Sanction Rate 

Sanctions imposed are recorded in the HAWI client file. The analysis was 
designed to determine whether members of experimental group had a higher probability 
of being sanctioned than members of the control group. As shown in Table 15, the 
estimated experimental effect is a positive and statistically significant 6.6 percent. This 
outcome may be attributed to the greater participation requirements imposed on the 
experimental group. Since the members of the control group were required to participate 
in fewer activities (most of them had no participation at all), they were not as much at 
risk for sanction for non-compliance with activity requirements, and therefore received 
fewer sanctions. 
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Table 15. Experimental Effects on Probability of Sanction 





Received Sanctions 
within 21 Months of 
Assignment 


Number of Subjects 


Experimental Group 




Number of Subjects 
Percent of Group 


193 

11.61% 


1662 

100.00% 


Control Group 




Number of Subjects 
Percent of Group 


79 

4.84% 


1632 

100.00% 










Unadjusted Net Experimental Effect 


6.77%*** 




Adjusted Net Experimental Effect (from 
Regression Analysis) 


6.59%*** 





Source: CSHR analysis of HDHS administrative data. 

Note: * * * indicates effects are statistically significant at a=O.01 level. 



Summary of Key Findings 

Differences in impacts between the experimental and control groups were 
calculated to judge the influence of JOBS WORKS! on participants’ self-sufficiency and 
AFDC participation. Specific self-sufficiency measures included: employment rates, 
length of employment, amount of total earnings, and total family income. AFDC 
measures included: average amount of AFDC benefits per case, AFDC exit and 
recidivism rates, use of subsidized child care, and rates at which persons were sanctioned 
for failing to comply with program requirements. 

Table 16 summarizes the results of these calculations. Taken together, these 
results indicate that the JOBS WORKS! demonstration significantly improved self- 
sufficiency for its early participants and reduced these persons’ dependence on AFDC. 
Rates of employment, length of employment, and total earnings were all significantly 
higher for experimental group members than control group members during the first 
twenty-one months after assignment. While the differences in total family income 
between the two groups were insignificant, this finding is not particularly meaningful 
because nearly every family in both groups had some income. Moreover, the amount of 
family income was significantly higher for experimental group members among those 
families with earned income. 
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Table 16. Summary of Findings 



Research Question 


Magnitude of 
estimated adjusted 
net effect 


Was estimated 
effect statistically 
significant? 


1 . Did the demonstration promote self-sufficiency? 






Employment Rate 


4.5 Percent 


Yes 


Length of Employment 


1.33 Months 


Yes 


Earnings 


28.5 Percent 


Yes 


Total Family Income 






Probability that family earned income 


0.04 Percent 


No 


Amount earned for those with earnings 


8.0 Percent 


Yes 


2. Did the demonstration affect AFDC participation? 






Average Per-Case Benefits 


-6.3 Percent 


Yes 


Exits 


5.2 Percent 


Yes 


Recidivism 


-1.58 Percent 


No 


Subsidized Child Care 






Probability that SCC was received 


-2.4 Percent 


Yes 


Amount of SCC for those who received it 


11.6 Percent 


No 


Sanction 


6.6 Percent 


Yes 



Source: CSHR analysis of HDHS administrative data. 



Experimental members also decreased their dependence on AFDC during the 
period of study. They experienced a 6.3 percent net decrease in their average AFDC 
benefits and left the rolls at significantly higher rates than control group members. 
However, the experiment had no significant effect on rates of AFDC recidivism, 
primarily due to the small sample size of persons returning to AFDC during this period. 
As expected, a significantly higher percentage of JOBS WORKS! participants received 
sanctions for non-participation. This occurred because experimental group members 
were required to participate in job search activities while most control group members 
were merely waiting to be served with no participation obligations. Child care usage was 
quite small for both groups, ranging from 4-6 percent of total AFDC recipients. 
Experimental group members used significantly less child care than control group 
members. These reasons for this finding are unclear. 

In conclusion, the JOBS WORKS! demonstration achieved most of its major 
objectives for early participants in the program. Although these early impacts are very 
positive, the shortened period of this evaluation only allowed impacts to be measured for 



twenty-one months after assignment to an experimental or control group. Thus, it is not 
possible to predict whether these findings will hold up for a longer period of time for 
these participants or whether similar positive results will occur for later groups of 
participants. Even so, this approach shows great promise for its current status as one 
component of the Hawaii JOBS program. 
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Appendix A 

Statistical Methods and Detailed Results 



Data Sources 

Table A-l summarizes the data used for this analysis and shows the periods of 
time for which data from each source was available. The waiver started in January 1995 
and ended in December 1996. In order to analyze long-term outcomes such as job 
retention and recidivism, it was necessary to collect data for a follow-up period after the 
end of the demonstration’s completion date. In addition to data for the demonstration and 
follow up period, data for some variables during the pre-demonstration period was 
collected in order to verify that the attributes of the experimental and control groups were 
equivalent before the starting of the experiment. 



Table A-l. Data Sources and Periods of Availability 



Child Care 
HANA (JOBS files) 
UI Wage and W4 



1994 


1995 


1996 


1997 


01 02 03 04 


01 02 03 04 


01 02 03 04 


01 02 03 04 


























^ — Pre-Demo 


Demonstration ^ 


Follow-up — ^ 



HAWI Data 

The HAWI data system contains most of the data needed to administer means- 
tested entitlement programs in Hawaii. It consists of five main files: a client file, a 
participation file, a benefits file, an income file and an address file. The client file 
provides demographic information such as age, race, educational attainment, and English 
language proficiency for clients. The participation file contains information on the dates 
of participation by clients in means-tested programs. The benefits file contains data on 
the amount of AFDC benefits paid to clients. The income file shows total family income 
attributed to each AFDC case at the time of application or re-certification for benefits. 
The address file shows current and past addresses for clients. CSHR received extractions 
from these files which covered all available records for all individuals in either the 
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experimental or control group in the waiver. This data source was used to provide 
information on duration, timing, and amount of AFDC benefits, AFDC exit rates, client 
and family demographics, and sanctions. 

JOBS Data 

Data for the JOBS program is collected in the HANA data system. These data 
include timing and amount of effort spent in JOBS component activities for participants 
in both regular JOBS and JOBS WORKS!. For individuals who have reported job entry, 
the HANA system also collects data on wages and duration of the employment. 

Support Services Data 

The research question regarding the use of child care as a support service was 
answered by analyzing data from the C 2 C child care data system. Data provided by 
HDHS included the date and dollar amount of all reimbursement checks issued to 
members of the experimental and control groups 

Employment and Earnings Data 

The HANA system provides self-reported employment data for JOBS 
participants. However, since most members of the control group never entered the JOBS 
program, the employment data in the HANA file will not include employment for most 
members of the control group. Thus, the HANA employment data is biased toward 
showing better employment outcomes for the experimental group, and may not be used 
for analysis . 8 

Past research has shown that data from the Unemployment Insurance system 
administered by Hawaii Department of Labor and Industrial Relations (DLER) has many 
potential advantages over self-reported employment data. The main advantage of the UI 
data is that it covers the control and experimental groups evenly. Any differences in 
employment observed from this source can be safely attributed to the experiment, rather 



8 Exploratory analysis of the hours and wage data contained in the HANA system was performed even 
though the data were known to be biased. While this analysis showed a strong effect on the probability of 
employment for the experimental group (a finding that is due to the bias) no particular differences in hours 
worked or wage rates were observed. 



than to differences in data collection procedures. This data source also has the advantage 
of being available throughout the full period of the waiver, as well as during the pre- 
demonstration and follow-up periods. U1 earnings and W4 (new hire) data from DLIR 
were merged with the HDHS data sources by SSN. 

Given the many advantages of the U1 earnings data, it is most regrettable that they 
contain only quarterly observations on earnings. The data do not contain exact start and 
end dates for the spells of employment, nor do they specify the number of hours worked 
or the wage rate. To facilitate computations, when analyzing the duration of employment 
we made the assumption that an individual was employed for all months contained in a 
quarter for which wages were reported. We further assumed that the reported quarterly 
wages were evenly divided between the months of the quarter. Hourly wage rates and 
hours of work may not be analyzed at all using U1 earnings data since it reports only the 
total quarterly earnings, and not the number of hours worked or the wage rate. Because 
of this data gap, a part time placement at a high wage can not be differentiated from a full 
time placement at a lower wage. 

JOBS WORKS! Administrative Records 

In addition to the usual data available from HDHS administrative files, HDHS 
provided a complete roster of all members of both the experimental and control groups. 
This roster included variables indicating the group assignment of the subject and the date 
the subject was selected for the evaluation. Observations in this dataset were used to 
select and analyze data from all other datasets, based on client numbers, case numbers, 
and social security numbers, as appropriate. 

Summary of Variables 

Table A-2 summarizes the variables used in the analysis. 
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Table A-2. Summary of Variables Used 



EXPRMT 


Dummy variable equal to one if subject was in experimental group, zero if in control group. 


HS 


Dummy variable equal to one if subject has highest grade completed of 12 or larger. 


MALE 


Dummy variable equal to one if subject was male, zero if female. 


CAT05 


Dummy variable equal to one if subject was an exempt individual waiting to volunteer for 
JOBS, zero otherwise 


CAT30 


Dummy variable equal to one if subject was non-exempt not in a target group, zero otherwise 


CAT65 


Dummy variable equal to one if subject was a JOBS volunteer in a target group, zero 
otherwise 


USCITZN 


Dummy variable equal to one if subject was a U.S. citizen, zero otherwise. 


FILIP 


Dummy variable equal to one if subject was of Philippine ethnicity, zero otherwise. 


HAWAITN 


Dummy variable equal to one if subject was of Hawaiian ethnicity, zero otherwise. 


WHITE 


Dummy variable equal to one if subject was of White ethnicity, zero otherwise. 


MIXED 


Dummy variable equal to one if subject was of Mixed ethnicity, zero otherwise. 


GOOD ENG 


Dummy variable equal to one if subject claimed English proficiency, zero otherwise. 


IN TG 


Dummy variable equal to one if subject was in a target group, zero otherwise. 


MARRIED 


Dummy variable equal to one if subject was married at time of selection, zero otherwise 


NEVMAR 


Dummy variable equal to one if subject had never been married at time of selection, zero 
otherwise 


NOTPRIM 


Dummy variable equal to one if subject was not the primary individual on the case and was not 
the sdousc of the primary individual, zero otherwise. 


VOLUNT 


Dummv variable equal to one if subject was a volunteer for JOBS, zero otherwise 


UI_EMPL 


Dummy variable equal to one if subject was employed within twenty one months of being 
selected. Based on UI wage file, zero otlrerwise. 


UI_WAGES 


Amount of covered UI wages from employment within twenty one months of being selected, 
for subjects who were employed within twenty one months of being selected. Not defined 
if the person was not employed. Based on UI wage file. 


PRE EMPL 


Amount of covered UI wages for the year prior to being selected. Based on UI wage file. 


HASINC 


Dummy variable equal to one if subject’s family had positive income within twenty one 
months of being selected, zero otherwise. Defined only for individuals whose families 
were either not in the experiment or who were in the same group as subject Not defined 
for subjects in families in which family members were in both experimental and control 
groups. 


TOTINC 


Amount of family income for individuals who had family income. Based on HAWI income 
file. Sum of all income found within 21 months of being selected. 


LNINC 


Natural logarithm of TOTINC 


BEN 


Amount of APDC benefits received by subject’s case for twenty one months after selectioa 
Based on HAWI benefits file. 


EXIT 


Dummy variable equal to one if individual exited Ah DC for more than 45 days within twenty 
one months of being selected, zero otherwise. * 


RECID 


Dummy variable defined only for individuals who exited, equal to one if the individual exited 
and returned to the rolls within twenty one months of selection, zero otherwise. 


CCUSED 


Dummy variable equal to one if the individual received subsidized child care within fifteen 
months after being selected, zero otherwise. 


TOTCC 


Dollars of subsidy for child care received within fifteen months of selection 


SANCED 


Dummy variable equal to one if subject received a sanction within twenty one months of being 
selected, zero otherwise. 
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Regression Analysis 

Multiple regression was applied to measure the adjusted net effects of the 
experiment. Table A-3 summarizes the regression equations that were fitted to answer 
each research question. Tables A-4 through A-14 show the regression results. 



Table A-3. Regression Models 



Research Question 


Dependent Variable 


Universe 


Did individuals in the experimental group 
have a higher probability of being 
employed than those in the control 
group? 


Dummy variable, 1 if 

employed during or after 
induction, zero otherwise 


All individuals in control or 
experimental groups 


Did individuals in the experimental group 
have a longer average length of 
employment than those in the control 
group? 


Average length of 

employment, or logarithm 
of average length of 
employment 


Employed individuals in 
control or experimental 
groups 


Did individuals in the experimental group 
have a higher level of earnings than 
those in the control group? 


Quarterly UI earnings or self- 
reported wage multiplied 
by self-reported hours 
from JOBS files, or the 
logarithm of these 
measures 


Employed individuals in 
control or experimental 
groups 


Did the families of the individuals in the 
experimental group have larger total 
family incomes than those in the 
control group? 


Family income 


Individuals in uncontami- 
nated cases with family 
income 


Was the average per-case level of benefits 
paid to the experimental group lower 
than the level for the control group? 


Total case benefits paid for 
participant's case. 
(Pre/Post change in 
benefits per case) 


All individuals in control or 
experimental groups 


Did individuals in the experimental group 
have a higher probability of exiting 
AFDC than individuals in the control 
group? 


Dummy variable, 1 if 
individual exits AFDC, 
zero otherwise 


All individuals in control or 
experimental groups 


Was the probability of returning to the 
rolls lower for former members of the 
experimental group than for former 
members of the control group? 


Dummy variable, 1 if 
individual returns to 
AFDC within follow-up, 
zero otherwise 


All individuals in control or 
experimental groups who 
exited. 


Did members of the experimental group 
have a higher probability of utilization 
of child care than members of the 
control group? 


Dummy variable, 1 if child 
care was used, zero 
otherwise. Also to be 
studied, total number of 
days of child care used. 


Separate regressions for: (1) 
all individuals (2) 
Individuals that received 
subsidized child care 


Was the sanction rate significantly 

different between the experimental and 
control groups? 


Dummy variable, 1 if 
sanction was imposed 
during or after 
participation, zero 
otherwise. 


All individuals in control or 
experimental groups. 
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Table A-4. Probability of Employment 



Regression Summary Statistics 



Dependent Mean 


0.51457 


R- Squared 


0.193 


Number of observations: 


3,294 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


0.4812 


4.329 


EXPRMT 


0.0449 


2.858 


HS 


0.0738 


4.208 


MALE 


0.0218 


0.966 


AGE_I 


-0.0064 


-5.985 


CAT05 


-0.1720 


-1.946 


CAT30 


-0.0493 


-0.508 


USCITZN 


0.0253 


0.78 


FILIP 


0.0490 


1.68 


HAWAIIN 


0.0034 


0.146 


WHITE 


0.0160 


0.554 


MIXED 


0.0182 


0.75 


GOODENG 


0.0179 


0.549 


INTG 


-0.0226 


-0.237 


MARRIED 


0.0157 


0.734 


NEVMAR 


-0.0074 


-0.364 


NOTPRIM 


0.0303 


0.767 


VOLUNT 


0.0647 


3.915 


PRE_EMPL 


0.4151 


24.011 




Table A-5. Regression for Length of Employment for 
Subjects Who Found Employment 



Regression Summary Statistics 



Dependent Mean 


12.15112 


R- Squared 


0.101 


Number of observations: 


1,694 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


3.5248 


1.361 


EXPRMT 


1.3298 


4.08 


HS 


0.8229 


2.183 


MALE 


-1.1698 


-2.542 


AGEI 


0.0619 


2.624 


CAT05 


2.3291 


1.087 


CAT30 


2.8063 


1.203 


USCITZN 


-0.1630 


-0.223 


FILIP 


1.2614 


2.168 


HAWAIIN 


-0.3401 


-0.704 


WHITE 


-1.2036 


-2.011 


MIXED 


0.5541 


1.11 


GOOD_ENG 


0.5666 


0.78 


IN_TG 


2.9348 


1.273 


MARRIED 


0.8078 


1.805 


NEVMAR 


0.3843 


0.913 


NOTPRIM 


0.8998 


1.208 


VOLUNT 


0.1666 


0.495 


PRE_EMPL 


3.7681 


11.399 





33 



Table A-6. Regression for Log of UI Wages for 
Subjects Who Found Employment 



Regression Summary Statistics 



Dependent Mean 


8.21858 


R- Squared 


0.068 


Number of observations: 


1,694 



Regression Coefficients 



Regressor 


Coefficient 


t-Statisdc 


INTERCEP 


6.2283 


9.861 


EXPRMT 


0.2507 


3.154 


HS 


0.3710 


4.036 


MALE 


0.2064 


1.839 


AGE_I 


0.0087 


1.512 


CAT05 


0.6937 


1.328 


CAT30 


0.8371 


1.471 


USC1TZN 


-0.1919 


-1.075 


FILIP 


0.3035 


2.139 


HAWAIIN 


-0.0253 


-0.214 


WHITE 


-0.1190 


-0.815 


MIXED 


0.1825 


1.498 


GOOD_ENG 


0.2732 


1.542 


IN_TG 


0.8391 


1.493 


MARRIED 


0.1113 


1.02 


NEVMAR 


-0.0813 


-0.792 


NOTPRIM 


0.0901 


0.496 


VOLUNT 


0.0252 


0.307 


PRE_EMPL 


0.5518 


6.844 
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Table A-7. Regression for Presence of Family Income 



Regression Summary Statistics 



Dependent Mean 


0.92448 


R-Squared 


0.131 


Number of observations: 


3.244 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


0.9647 


15.598 


EXPRMT 


0.0004 


0.05 


HS 


-0.0086 


-0.893 


MALE 


-0.1353 


-10.82 


AGE_I 


0.0000 


- 0.074 


CAT05 


-0.0194 


-0.392 


CAT30 


-0.0194 


-0.358 


USCITZN 


0.0603 


3.338 


FILIP 


0.0121 


0.753 


HAWAIIN 


0.0120 


0.933 


WHITE 


-0.0237 


-1.481 


MIXED 


0.0020 


0.146 


GOOD_ENG 


-0.0181 


-0.999 


IN_TG 


-0.0135 


-0.255 


MARRIED 


-0.0761 


-6.4 


NEVMAR 


-0.0085 


-0.762 


NOTPRIM 


-0.1883 


-8.517 


VOLUNT 


0.0088 


0.962 
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Table A-8. Regression for Logarithm of Family Income for 
Those Who Had Family Income 



Regression Summary Statistics 



Dependent Mean 


9.47524 


R- Squared 


0.050 


Number of observations: 


2,999 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


9.1211 


40.193 


EXPRMT 


0.0770 


2.376 


HS 


0.0594 


1.651 


MALE 


-0.2867 


-5.924 


AGEI 


0.0046 


2.043 


CAT05 


0.1900 


1.053 


CAT30 


0.0267 


0.135 


USCITZN 


0.1712 


2.492 


FILIP 


0.0471 


0.789 


HAWAHN 


0.0727 


1.524 


WHITE 


-0.0975 


-1.638 


MIXED 


0.0394 


0.788 


GOOD_ENG 


-0.0006 


-0.009 


IN_TG 


0.0372 


0.191 


MARRIED 


0.0371 


0.832 


NEVMAR 


-0.0739 


-1.817 


NOTPRIM 


-0.5559 


-5.877 


VOLUNT 


-0.0101 


-0.296 
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Table A-9* Regression for Amount of AFDC Benefits Received 



Regression Summary Statistics 



Dependent Mean 


11,150 


R-Squared 


0.036 


Number of observations: 


3,244 



Regression Coefficients 



Regressor 


CoefFicient 


t-Statistic 


INTERCEP 


11075.0000 


6.778 


EXPRMT 


-702.3470 


-3.057 


HS 


1091.1851 


-4.268 


MALE 


-400.4039 


-1.212 


AGE_I 


26.7739 


1.691 


CAT05 


897.1012 


0.687 


CAT30 


-934.7935 


-0.654 


USCITZN 


1557.9761 


3.264 


FILIP 


1149.4876 


-2.711 


HAWAHN 


851.6686 


2.51 


WHITE 


-955.5805 


-2.263 


MIXED 


-163.7751 


-0.461 


GOOD_ENG 


-806.7504 


-1.689 


EN_TG 


-451.4417 


-0.321 


MARRIED 


1185.1990 


3.775 


NEVMAR 


-185.9624 


-0.633 


NOTPRIM 


1604.0978 


2.746 


VOLUNT 


227.4419 


0.942 
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Table A-10. Regression for AFDC Exits 



Regression Summary Statistics 



Dependent Mean 


0.4915 


R- Squared 


0.044 


Number of observations: 


3,294 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


0.6495 


5.37 


EXPRMT 


0.0521 


3.043 


HS 


0.0526 


2.76 


MALE 


0.1159 


4.737 


AGEI 


-0.0040 


-3.418 


CAT05 


-0.1452 


-1.509 


CAT30 


-0.0615 


-0.582 


uscrrzN 


-0.1198 


-3.395 


FILIP 


0.0544 


1.716 


HAWAIIN 


-0.0732 


-2.899 


WHITE 


0.0290 


0.921 


MIXED 


-0.0068 


-0.257 


GOODENG 


00977 


2.752 


IN_TG 


-0.0558 


-0.537 


MARRIED 


-0.0184 


-0.791 


NEVMAR 


-0.0833 


- 3.792 


NOTPRIM 


0.1505 


3.493 


VOLUNT 


0.0245 


1.361 
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Table A-ll. Regression for Recidivism for Those Who Exited 



Regression Summary Statistics 



Dependent Mean 


0.31007 


R-Squared 


0.027 


Number of observations: 


1,619 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


0.0569 


0.31 


EXPRMT 


-0.0158 


-0.691 


HS 


0.0147 


0.562 


MALE 


-0.0005 


-0.016 


AGE_I 


-0.0056 


-3.635 


CAT05 


0.3304 


2.128 


CAT30 


0.3766 


2.243 


USCITZN 


0.0739 


1.662 


FILIP 


0.0350 


0.878 


HAWAI1N 


0.0457 


1.332 


WHITE 


0.0047 


0.115 


MIXED 


0.0038 


0.109 


GOODENG 


-0.0585 


-1.197 


IN_TG 


0.3861 


2.328 


MARRIED 


0.0934 


3.104 


NEVMAR 


-0.0295 


-0.988 


NOTPRIM 


-0.0474 


-0.937 


VOLUNT 


0.0227 


0.949 
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Table A-12. Regression for Whether Client Received Child Care 



Regression Summary Statistics 



Dependent Mean 


0.05464 


R-Squared 


0.041 


Number of observations: 


3,294 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


0.0963 


1.749 


EXPRMT 


-0.0243 


-3.111 


HS 


0.0051 


0.589 


MALE 


-0.0265 


-2.383 


AGEI 


-0.0016 


-2.959 


CAT05 


-0.0571 


-1.304 


CAT30 


0.0033 


0.07 


USCITZN 


0.0076 


0.472 


FILIP 


0.0214 


1.481 


HAWAIIN 


0.0021 


0.182 


WHITE 


0.0121 


0.847 


MIXED 


0.0229 


1.897 


GOOD_ENG 


-0.0115 


-0.714 


INTG 


0.0643 


1.359 


MARRIED 


-0.0228 


-2.156 


NEVMAR 


-0.0096 


-0.964 


NOTPRIM 


-0.0456 


-2.323 


VOLUNT 


0.0243 


2.968 
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Table A- 13 . Regression for Amount of Subsidized Child Care Received for 
Individuals Who Received Child Care 



Regression Summary Statistics 



Depeixient Mean 


1593.37 


R-Squared 


0.079 


Number of observations: 


180 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


3098.6265 


1.083 


EXPRMT 


115.3034 


0.347 


HS 


-32.9046 


-0.089 


MALE 


-481.5012 


-0.64 


AGEI 


-42.9519 


-1.681 


CAT05 


-2103.6267 


-1.334 


CAT30 


-565.0403 


-0.21 


uscrrzN 


-8.6861 


-0.011 


FILIP 


765.3694 


1.309 


HAWAIIN 


410.5301 


0.8 


WHITE 


431.1029 


0.732 


MIXED 


576.9733 


1.188 


GOODENG 


-444.3508 


-0.691 


IN_TG 


73.4153 


0.028 


MARRIED 


-505.7812 


-1.014 


NEVMAR 


-264.4850 


-0.697 


NOTPRIM 


0.0000 


see note 


VOLUNT 


353.8100 


1.019 



Note: NOTPRIM is dropped from this regression because all 

individuals in the regression were primary individuals on their 
cases. 
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Table A-14. Regression for Sanctions Imposed 



Regression Summary Statistics 



Dependent Mean 


0.08257 


R-Squared 


0.027 


Number of observations: 


3,294 



Regression Coefficients 



Regressor 


Coefficient 


t-Statistic 


INTERCEP 


0.0546 


0.813 


EXPRMT 


0.0659 


6.936 


HS 


0.0029 


0.273 


MALE 


0.0317 


2.333 


AGE_I 


-0.0016 


-2.519 


CAT05 


-0.0682 


-1.276 


CAT30 


0.0032 


0.055 


uscrrzN 


0.0310 


1.58 


FILIP 


0.0013 


0.073 


HAW AON 


0.0149 


1.061 


WHITE 


0.0112 


0.643 


MIXED 


0.0230 


1.567 


GOOD_ENG 


-0.0091 


-0.463 


IN_TG 


0.0203 


0.353 


MARRIED 


-0.0109 


-0.845 


NEVMAR 


-0.0121 


-0.993 


NOTPRIM 


0.0069 


0.287 


VOLUNT 


0.0225 


2.252 
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